{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NAML: Neural News Recommendation with Attentive Multi-View Learning\n",
    "NAML \\[1\\] is a multi-view news recommendation approach. The core of NAML is a news encoder and a user encoder. The newsencoder is composed of a title encoder, a body encoder, a vert encoder and a subvert encoder. The CNN-based title encoder and body encoder learn title and body representations by capturing words semantic information. After getting news title, body, vert and subvert representations, an attention network is used to aggregate those vectors. In the user encoder, we learn representations of users from their browsed news. Besides, we apply additive attention to learn more informative news and user representations by selecting important words and news.\n",
    "\n",
    "## Properties of NAML:\n",
    "- NAML is a multi-view neural news recommendation approach.\n",
    "- It uses news title, news body, news vert and news subvert to get news repersentations. And it uses user historical behaviors to learn user representations.\n",
    "- NAML uses additive attention to learn informative news and user representations by selecting important words and news.\n",
    "\n",
    "## Data format:\n",
    "\n",
    "### train data\n",
    "One simple example: <br>\n",
    "\n",
    "`1 0 0 0 0 Impression:0 User:502 CandidateTitle0:17917,36557,47926,32224,24113,48923,19086,5636,3703,0... CandidateBody0:17024,53305,8832,29800,9787,4068,48731,48923,19086,38699,5766,22487,38336,29800,8548,39128,33457,7789,\n",
    "30543,7482,8548,49004,53305,22999,32747,21103,11799,5766,4868,17115,7482,15118,48731,2025,7789,23336,7789,48731,19086,\n",
    "10630,11128,36557,3703,47354,611,7789,19086,5636,51521,30706... CandidateVert0:14... CandidateSubvert0:219... ClickedTitle0:48,33405,35198,5969,5636,35845,850,48731,46799,24113... ClickedBody0:36557,67,34519,24113,8548,48,33405,35198,5969,14340,7053,850,8823,9498,46799,24113,12506,32747,31130,\n",
    "3074,48731,20869,14264,38289,37310,7789,36557,34967,48731,36916,23321,3595,48731,47354,4868,15719,7482,12771,50693,\n",
    "47354,17523,48,20918,17900,35198,48731,20869,1220,14264,7789... ClickedVert0:14... ClickedSubvert0:99... `\n",
    "<br>\n",
    "\n",
    "In general, each line in data file represents one positive instance and n negative instances in a same impression. The format is like: <br>\n",
    "\n",
    "`[label0] ... [labeln] [Impression:i] [User:u] [CandidateTitle0:w1,w2,w3,...] ... [CandidateBody0:w1,w2 ..] ... [CandidateVert0:v] ... [CandidateSubvert0:s] ... [ClickedTitle0:w1,w2,w3,...] ... [ClickedBody0:w1,w2,w3,...] ... [ClickedVert0:v] ... [ClickedSubvert0:s] ...`\n",
    "\n",
    "<br>\n",
    "\n",
    "It contains several parts seperated by space, i.e. label part, Impression part `<impresison id>`, User part `<user id>`, CandidateNews part, ClickedHistory part. CandidateNews part describes the target news article we are going to score in this instance. It is represented by (aligned) title words, body words, news vertical index and subvertical index. To take a quick example, a news title may be : `Trump to deliver State of the Union address next week` , then the title words value may be `CandidateTitlei:34,45,334,23,12,987,3456,111,456,432`. <br>\n",
    "ClickedHistory describe the k-th news article the user ever clicked and the format is the same as candidate news. Every clicked news has title words, body words, vertical and subvertical. We use a fixed length to describe an article, if the title or body is less than the fixed length, just pad it with zeros.\n",
    "\n",
    "### test data\n",
    "One simple example: <br>\n",
    "`1 Impression:0 User:1529 CandidateTitle0:5327,18658,13846,6439,611,50611,0,0,0,0 CandidateBody0:13846,3197,27902,225,5327,45008,29145,7789,509,7395,11502,36557,13846,23680,26492,38072,20507,5636,\n",
    "4247,32747,50132,7482,41049,32747,43022,50611,35979,7789,1191,36557,52870,21622,48148,42737,48731,36557,13846,23680,\n",
    "13173,7482,13848,38072,20507,7789,41675,36875,51461,12348,21045,42160 CandidateVert0:14 CandidateSubvert0:19 ClickedTitle0:9079,3703,32747,8546,19377,50184,32747,24026,40010,49754 ... ClickedBody0:26061,48731,8576,7789,8683,9079,5636,45084,46805,3703,509,43036,11502,28883,9498,18450,32747,8546,33405,\n",
    "35647,50184,7482,41143,8220,43618,38072,35198,43390,28057,32552,45245,10764,16247,4221,41038,36557,43683,46805,7789,\n",
    "29727,2179,51003,34797,897,21045,12974,23382,46287,48731,15206 ... ClickedVert0:14 ... ClickedSubvert0:219 ...`\n",
    "<br>\n",
    "\n",
    "In general, each line in data file represents one instance. The format is like: <br>\n",
    "\n",
    "`[label] [Impression:i] [User:u] [CandidateTitle0:w1,w2,w3,...] [CandidateBody0:w1,w2,w3,...] [CandidateVert0:v] [CandidateSubvert0:s] [ClickedTitle0:w1,w2,w3,...] ... [ClickedBody0:w1,w2,w3,...] ... [ClickedVert0:v] ... [ClickedSubvert0:s] ...`\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global settings and imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21) \n",
      "[GCC 7.3.0]\n",
      "Tensorflow version: 1.15.2\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "from reco_utils.recommender.deeprec.deeprec_utils import download_deeprec_resources \n",
    "from reco_utils.recommender.newsrec.newsrec_utils import prepare_hparams\n",
    "from reco_utils.recommender.newsrec.models.naml import NAMLModel\n",
    "from reco_utils.recommender.newsrec.io.naml_iterator import NAMLIterator\n",
    "import papermill as pm\n",
    "from tempfile import TemporaryDirectory\n",
    "import tensorflow as tf\n",
    "import os\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Tensorflow version: {}\".format(tf.__version__))\n",
    "\n",
    "tmpdir = TemporaryDirectory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 72.6k/72.6k [00:04<00:00, 16.7kKB/s]\n"
     ]
    }
   ],
   "source": [
    "data_path = tmpdir.name\n",
    "yaml_file = os.path.join(data_path, r'naml.yaml')\n",
    "train_file = os.path.join(data_path, r'train.txt')\n",
    "valid_file = os.path.join(data_path, r'test.txt')\n",
    "wordEmb_file = os.path.join(data_path, r'embedding.npy')\n",
    "if not os.path.exists(yaml_file):\n",
    "    download_deeprec_resources(r'https://recodatasets.blob.core.windows.net/newsrec/', data_path, 'naml.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "epochs=5\n",
    "seed=42"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:\n",
      "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "  * https://github.com/tensorflow/io (for I/O related ops)\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "data_format=naml,iterator_type=None,wordEmb_file=/tmp/tmp37nn3h_g/embedding.npy,doc_size=None,title_size=10,body_size=50,word_emb_dim=100,word_size=54071,user_num=10329,vert_num=17,subvert_num=249,his_size=50,npratio=4,dropout=0.2,attention_hidden_dim=100,head_num=4,head_dim=100,cnn_activation=relu,dense_activation=relu,filter_num=200,window_size=3,vert_emb_dim=100,subvert_emb_dim=100,gru_unit=400,type=ini,user_emb_dim=50,learning_rate=0.0001,loss=cross_entropy_loss,optimizer=adam,epochs=5,batch_size=64,show_step=100000,metrics=['group_auc', 'mean_mrr', 'ndcg@5;10']\n"
     ]
    }
   ],
   "source": [
    "hparams = prepare_hparams(yaml_file, wordEmb_file=wordEmb_file, epochs=epochs)\n",
    "print(hparams)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterator = NAMLIterator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the NAML model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From ../../reco_utils/recommender/newsrec/models/base_model.py:39: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
      "\n",
      "WARNING:tensorflow:From /data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "WARNING:tensorflow:From ../../reco_utils/recommender/newsrec/models/base_model.py:54: The name tf.GPUOptions is deprecated. Please use tf.compat.v1.GPUOptions instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model = NAMLModel(hparams, iterator, seed=seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From ../../reco_utils/recommender/newsrec/io/naml_iterator.py:135: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
      "\n",
      "{'group_auc': 0.533, 'mean_mrr': 0.1792, 'ndcg@5': 0.177, 'ndcg@10': 0.2417}\n"
     ]
    }
   ],
   "source": [
    "print(model.run_eval(valid_file))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "at epoch 1\n",
      "train info: logloss loss:1.5924944522429485\n",
      "eval info: group_auc:0.5565, mean_mrr:0.1811, ndcg@10:0.2487, ndcg@5:0.1817\n",
      "at epoch 1 , train time: 46.0 eval time: 38.3\n",
      "at epoch 2\n",
      "train info: logloss loss:1.5695580341378037\n",
      "eval info: group_auc:0.5568, mean_mrr:0.1817, ndcg@10:0.2475, ndcg@5:0.1769\n",
      "at epoch 2 , train time: 42.9 eval time: 38.1\n",
      "at epoch 3\n",
      "train info: logloss loss:1.5507925962915226\n",
      "eval info: group_auc:0.5633, mean_mrr:0.1884, ndcg@10:0.2528, ndcg@5:0.1943\n",
      "at epoch 3 , train time: 42.9 eval time: 38.2\n",
      "at epoch 4\n",
      "train info: logloss loss:1.5379242575898462\n",
      "eval info: group_auc:0.5661, mean_mrr:0.1933, ndcg@10:0.256, ndcg@5:0.1952\n",
      "at epoch 4 , train time: 42.8 eval time: 38.2\n",
      "at epoch 5\n",
      "train info: logloss loss:1.5233123536012612\n",
      "eval info: group_auc:0.5666, mean_mrr:0.1919, ndcg@10:0.2574, ndcg@5:0.1955\n",
      "at epoch 5 , train time: 42.9 eval time: 38.3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<reco_utils.recommender.newsrec.models.naml.NAMLModel at 0x7f8f701fef60>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_file, valid_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'group_auc': 0.5666, 'mean_mrr': 0.1919, 'ndcg@5': 0.1955, 'ndcg@10': 0.2574}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "application/papermill.record+json": {
       "res_syn": {
        "group_auc": 0.5666,
        "mean_mrr": 0.1919,
        "ndcg@10": 0.2574,
        "ndcg@5": 0.1955
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res_syn = model.run_eval(valid_file)\n",
    "print(res_syn)\n",
    "pm.record(\"res_syn\", res_syn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\\[1\\] Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang and Xing Xie: Neural News Recommendation with Attentive Multi-View Learning, IJCAI 2019<br>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python (reco_gpu)\n",
   "language": "python",
   "name": "reco_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
